On May 9, 2017, the U.S. Department of the Treasury released the first batch of spending data that U.S. agencies submitted under the DATA Act, a 2014 law that requires:
This is what you might call a BFD.
If you don't want to follow these meanderings around the new beta.usaspending.gov website, here are the quick takeaways:
To understand magnitude of the May 9 release, consider this diagram lifted from the new beta.usaspending.gov website about what you can see there:
Here's the part of the spending hierarchy that you can see on the pre-DATA Act USAspending.gov website:
Other parts were available in bits and pieces but weren't public, searchable, and machine-readble. Furthermore, those bits and pieces didn't link to other parts of the hierachy, making it impossible to get the full picture.
Nothing against awards data. It's useful. Awards describe grants, contracts, and loans. This category of federal spending also includes information about payments the U.S. government makes on behalf of individuals, like Social Security and Medicare payments. Awards are a terrific way to understand how our collective tax dollars flow back into our communities.
For example, here's a grant that Philadelphia, PA received from the Environmental Protection Agency to provide assistance to the City of Philadelphia in its efforts to implement air pollution control programs.
Interesting! But not new. We can see this type of information on the legacy usaspending.gov website.
Also! Awards data comes from the day-do-day systems that agencies use to manage their contracts, grants, loans, etc. It does not represent data from an audited financial system.
The DATA Act achieves what fedspending data nerds sometimes call the "golden spike": the linkage of an award back to the federal account that funds it. See the Financial System Details tab down there at the bottom of the award page? That's the golden spike.
Notice this sentence: It comes directly from the audited financial systems of federal agencies.
Now we see can see that the reported \$1,000,000 grant amount matches the \$1,000,000 amount being reported by the EPA's audited financial system.
Furthermore, the Financial Systems Details view of the awards data (shown above) gives us the linkages we need to trace an award up to the level immediately above it in the spending hierarchy:
Categorical Grant: State and Local Air Quality ManagementGrants, subsidies, and contributionsProgram activity and object class are two separate (but sometimes complementary) ways of categorizing the spending information stored in agency financial systems. Let's consider them in the context of that EPA grant to Philadelphia.
The object class isn't super interesting, because we already know we're looking at a grant. What is interesting is program activity. We can see that this grant is part of a larger bucket of spending for a program called State and Local Air Quality.
This is already more information than we could have extracted from the current usaspending.gov, which doesn't provide any way to understand an individual award in the context of the greater spending landscape.
Beta.usaspending.gov is in...beta. Thus, there's currently no easy way to look up everything related to a specific program activity. (hint: there's community forum where you can do your patriotic duty by suggesting improvements and features).
That said, you can get geeky and use the USAspending API to follow the trail. The API (application programming interface) is the behind-the-scenes way that the website retrieves data. Luckily, the API not only powers the website: it's also available for the public to query.
For example: this strange-looking link to the USAspending API requests a list of awards that belong to the State and Local Air Quality program activity. The code below shows how you can use the API to get the details of these awards and, as a result, a better sense of State and Local Air Quality spending.
In [3]:
# For those interested in the API, here's an example of displaying that list of
# all awards with the "State and Local Air Quality" program activity classification
import json
import requests
import pandas as pd
from pandas.io.json import json_normalize
uri = 'https://api.usaspending.gov/api/v1/accounts/awards/?program_activity__program_activity_name=Categorical%20Grant:%20%20State%20and%20Local%20Air%20Quality%20Management&limit=500'
r = requests.get(uri)
air_awards = pd.DataFrame(json_normalize(r.json()['results']))
# This is pretty good, but we can can show better info by using the award identifiers
# in the air quality award list to grab some additional details about each item
award_list = air_awards['award'].tolist()
uri = 'https://api.usaspending.gov/api/v1/awards/'
headers = {'content-type': 'application/json'}
payload = {
"limit": 500,
"fields": [
"id", "total_obligation", "description", "type", "type_description",
"fain", "piid", "uri", "recipient", "place_of_performance"],
"filters": [{"field": "id", "operation": "in", "value": award_list}]
}
r = requests.post(uri, data=json.dumps(payload), headers=headers)
air_award_details = pd.DataFrame(json_normalize(r.json()['results']))
air_merged = pd.merge(air_awards, air_award_details, left_on='award', right_on='id')
# as an example, just show the first 10 awards
air_merged[[
'type_description', 'recipient.recipient_name', 'recipient.location.city_name',
'program_activity.program_activity_code', 'program_activity.program_activity_name',
'treasury_account.federal_account.account_title'
]].head(10)
Out[3]:
We've now followed a single award up the federal spending chain to get a list of other awards that are funded from that same State and Local Air Quality program. Some of these awards are grants, similar to the one that went to Philly. Others are contracts that appear to be program-related services like air monitoring equipment and databases.
We had to dig into the API to do this, but I suspect the website itself will eventually support this type of sleuthing (the team releases new features bi-weekly).
The resulting list gives us a better picture of the overall reach of the program. In this first DATA Act reporting period, the EPA made ~150 State and Local Air Quality awards.
But again, awards spending is only part of the picture. Presumably there are other costs to running the State and Local Air Quality program. To find those, we have to go up another level in the spending hierarchy.
Loosely speaking, a federal account represents a high-level grouping of specific, related Treasury accounts. Simplistically, you can think of Treasury accounts as funds that correspond to Congress authorizations to spend money for specific purposes.
In the table of air quality grants that we got from the API, you can see that all of them are associated with a federal account titled State and Tribal Assistance Grants, Environmental Protection Agency.
Now that we know this title, we can look it up on the search page of beta.usaspending.api.
The result of that search will be a view of the money obligated for of this specific account: \$369,943,892, for fiscal year 2017 to date.
(An obligation means that money has been set aside for a designated purpose, to be paid later.)
It's not immediately obvious, but the search results page lets you jump right to the Federal Accounts level of the spending hiearchy. Look for the Spending by Budget Category chart and choose the Federal Accounts view to see spending by federal account.
In this case, you'll only see one account, because we only searched for one. What you want is that hyperlink on the y axis: https://beta.usaspending.gov/#/federal_account/1752
Click it for a high-level view of activity for the entire account: not just awards spending.
You're now looking at the Federal Account Summary page, which displays the details of an individual federal account:
Back to our original EPA air quality award to Philadelphia and the State and Tribal Assistance Grants, Environmental Protection Agency program that funds it.
The Federal Account Summary page lets you drill into a federal account by any combination of program activity codes and object classes. Thus, to better understand the overall spending for State and Tribal Assistance Grants, Environmental Protection Agency (not just grants), you would:
State and Tribal Assistance Grants, Environmental Protection Agency program activity (left-hand side of the screen)Now you have the high-level categories of spending for this program.
You can switch up the filters on the left-hand side of the screen to explore the other programs and object classes that compose the State and Tribal Assistance Grants, Environmental Protection Agency account. For example, filtering on the Peronnel Compensation and Benefits object class shows that less than 1% of this account's obligated funds go towards salaries and benefits.
Many of us view federal spending solely from the perspective of award data, because it's all we've had access to in a convenient, machine-readable format.
Before the DATA Act, the only thing we could see was the \$1,000,000 grant to City of Philadelphia. Post-DATA Act, we have the context that federal spending nerds have dreamed of for decades:
Natural Resources budget functionState and Tribal Assistance Grants, Environmental Protection Agency federal accountState and Local Air Quality programSome people will use this information to better understand how our tax dollars fund the services we use every day. Some will use it to look for duplication and waste. Some will use it to track spending for the programs they manage. Some will use it to surface areas for continued data improvement.
Whatever your angle, May 9, 2017 saw a huge step forward.